Information technology (IT) has become an integral part of most businesses, as accessing and managing information becomes more and more essential to compete in today's global marketplace. By having a robust IT infrastructure, a business can quickly and efficiently respond to customer demands and needs. Along with the increase in importance of IT, however, has come additional demand to store, maintain, and protect the data associated with a business, such as customer records, technical information or designs, or other business information. Storage of information, and proper backup of this information, has become an important challenge for IT managers. The proliferation of compression technologies (e.g., MP3, MPEG-4, DVI, etc.) and the explosion of data storage have fueled the rapid growth in storage capacity, causing IT or storage managers to look for ways to improve storage cost efficiencies.
Many companies, particularly larger companies or ones more dependent on their IT infrastructure, rely on a storage system that includes servers and storage devices connected with a local area network (LAN) or other network. Storage devices can include hard drive arrays, storage servers, RAID's (Redundant Array of Independent Disks), removable-medium storage devices (e.g., tape backup drives), or other devices to storage information. Information is then stored on one or more of the storage devices (depending on the level of safety required). Storage devices can be a very significant part of the cost of an IT infrastructure, particularly if information is duplicated on different devices to improve safety of the data. Accordingly, storage or IT managers typically desire to reduce the amount of storage necessary to meet the needs of the company's IT infrastructure and to thus improve storage cost efficiencies.
One common solution to the increase of data and the resulting increase in storage requirements is to compress the stored files, resulting in continually more advanced data compression techniques being developed to curb the growth in storage capacity. Current file compression techniques involve analyze a file or set of files for common patterns which can then be stored or referenced again multiple times so that with each subsequent reference to the stored pattern a significant amount of space is saved. With these technologies, a file or set of files that had more repeated data throughout could be compressed to a higher degree. Compression technologies such as WinRAR® by win.rar GmbH of Bremen, Germany (http://www.win-rar.com/) analyze several files at once and then apply compression over an entire fileset to provide a few percentage points of increased compression due to the larger data set with which the analysis can be made. This method has a number of disadvantages, such as the limitation that individual files cannot be decompressed without reading the entire compressed package resulting in slow extracts. Moreover, the file set must be stored as a single file and must be determined at compression time, prohibiting its use for subsequent compression sessions. There is, therefore, a need for an efficient and effective system to reduce the size of files in a storage system.